Harmonious, Hollow or Hostile: Exploring cognitive offloading in multimodal representations involving haptic and visual modalities

1. PROBLEM Tangible user interfaces (TUIs) have emerged as an alternative interface paradigm whereby the interface is comprised of one or more physical objects infixed with computation. Two fundamental aspects of TUIs are the coupling of external representation with physical artefacts and the multimodality inherent in tangibles. Price [6] has argued for greater emphasis on the role of representation, particularly with respect to cognition, in the design of tangible environments. Recent research reflects this need; Bakker et al.’s Weathergods addresses conceptual representation in tangibles, i.e. symbolic and iconic representations [2 & 3]. Echoing Price’s assertions, making use of the multimodality to offload cognition is supported by cognitive learning theories, for example Cognitive Load Theory [7]. Further, at least one study has shown the importance of considering cognitive load in the design of multimodal systems [4]. However, TUIs possess two modalities, visual and tactile (haptic), dealt with by the same cognitive system, namely the visual-spatial sketchpad [1]. This calls into question the effect of multimodal representation involving haptic and visual modalities on distributed cognitive load, which has not yet been empirically explored [1].